Skip to content

🐛 Concurrent S3 bucket creation attempt #8045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jul 7, 2025
Merged

🐛 Concurrent S3 bucket creation attempt #8045

merged 6 commits into from
Jul 7, 2025

Conversation

@giancarloromeo
Copy link
Contributor

@giancarloromeo giancarloromeo commented Jul 4, 2025 •
edited
Loading

What do these changes do?

This PR makes the setup of S3 in storage more robust, especially when trying to create the Bucket.

It would be ideal to move the creation out of the service, in a script, and not in the service logic.

Related issue/s

How to test

Dev-ops

@giancarloromeo giancarloromeo added this to the Engage milestone Jul 4, 2025
@giancarloromeo giancarloromeo self-assigned this Jul 4, 2025
@giancarloromeo giancarloromeo added the a:storage issue related to storage service label Jul 4, 2025
@codecov
Copy link

codecov bot commented Jul 4, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.30%. Comparing base (0fc43f7) to head (f6b5fef).

❗ There is a different number of reports uploaded between BASE (0fc43f7) and HEAD (f6b5fef). Click for more details.

HEAD has 31 uploads less than BASE
Flag BASE (0fc43f7) HEAD (f6b5fef)
unittests 32 1
Additional details and impacted files 
@@             Coverage Diff             @@
##           master    #8045       +/-   ##
===========================================
- Coverage   88.13%   68.30%   -19.83%     
===========================================
  Files        1811      753     -1058     
  Lines       69732    35235    -34497     
  Branches     1258      176     -1082     
===========================================
- Hits        61459    24068    -37391     
- Misses       7909    11109     +3200     
+ Partials      364       58      -306
Flag Coverage Δ
integrationtests 64.34% <ø> (+0.05%) ⬆️
unittests 86.39% <100.00%> (-0.33%) ⬇️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_celery_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 76.86% <ø> (-8.13%) ⬇️
agent ∅ <ø> (∅)
api_server ∅ <ø> (∅)
autoscaling ∅ <ø> (∅)
catalog ∅ <ø> (∅)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 77.77% <ø> (-13.18%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 88.33% <ø> (-1.77%) ⬇️
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage 86.39% <100.00%> (+0.02%) ⬆️
webclient ∅ <ø> (∅)
webserver 59.08% <ø> (-29.60%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fc43f7...f6b5fef. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@giancarloromeo giancarloromeo changed the title 🐛 Concurrent S3 bucket creation 🐛 Concurrent S3 bucket creation attempt Jul 4, 2025
@giancarloromeo giancarloromeo marked this pull request as ready for review July 4, 2025 10:18
@giancarloromeo giancarloromeo requested a review from sanderegg as a code owner July 4, 2025 10:18
sanderegg
sanderegg approved these changes Jul 4, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check my comments.

Also I do not agree with the "it should be done in a separate script". Why?

pcrespov
pcrespov approved these changes Jul 4, 2025
View reviewed changes
@giancarloromeo giancarloromeo enabled auto-merge (squash) July 7, 2025 06:45
@giancarloromeo giancarloromeo added the 🤖-automerge marks PR as ready to be merged for Mergify label Jul 7, 2025
@giancarloromeo
Copy link
Contributor Author

giancarloromeo commented Jul 7, 2025

@Mergifyio queue

@mergify
Copy link
Contributor

mergify bot commented Jul 7, 2025 •
edited
Loading

queue

🛑 The pull request has been removed from the queue default

The following conditions don't match anymore:

  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • any of: [🛡 GitHub branch protection]
        • check-neutral = deploy to dockerhub
        • check-skipped = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-neutral = system-tests
        • check-skipped = system-tests
        • check-success = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-neutral = integration-tests
        • check-skipped = integration-tests
        • check-success = integration-tests

@sonarqubecloud
Copy link

sonarqubecloud bot commented Jul 7, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@mergify
Copy link
Contributor

mergify bot commented Jul 7, 2025

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

@giancarloromeo giancarloromeo merged commit 7f873b9 into ITISFoundation:master Jul 7, 2025
146 of 152 checks passed
@giancarloromeo giancarloromeo deleted the is8043/fix-storage-s3-startup branch July 7, 2025 09:03
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Aug 5, 2025
Closed
88 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcrespov pcrespov pcrespov approved these changes

@sanderegg sanderegg sanderegg approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@bisgaard-itis bisgaard-itis Awaiting requested review from bisgaard-itis

@GitHK GitHK Awaiting requested review from GitHK

Assignees

@giancarloromeo giancarloromeo

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:storage issue related to storage service

Projects

None yet

Milestone

Engage

Development

Successfully merging this pull request may close these issues.

The sto-worker service fails to start due to concurrent S3 bucket creation during startup

4 participants

@giancarloromeo @pcrespov @sanderegg @matusdrobuliak66